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ABSTRACT 



The aim of the TEEODE project was to develop a 
representative survey of the models of assessment and evaluation used by 
institutions supporting distance learning in the 15 member states of the 
European Union. This paper focuses on issues of course development and 
assessment, and, in particular, on how effective learning outcomes are 
achieved and how the learning is communicated to the learner. The emphasis is 
on the tutors' perceptions, teaching assessment, and support-related issues. 
The introduction provides an overview of the project and a summary of "deep” 
and "surface” approaches to learning. Development of the Course Tutor 
Questionnaire and the survey sample (n=224) are then described; use of an 
online questionnaire widened the sample to include data from Israel, Mexico, 
and the United States. Results in the following areas are discussed: the 
tutors, the courses, the students, assessment of the courses, production of 
the assessment material, delivery of the assessment instruments, and staff 
development. Four tables present data on the subject areas of ODE (Open and 
Distance Education) courses, the level of courses, main objectives for 
assessing students, and areas in which advice and/or training would be 
appreciated by the tutors. (DLS ) 
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Introduction 

Distance learning courses are an important way of meeting society's needs for more flexible ways of educating and 
training a modem work force. Supporting students at a distance raises new and interesting questions. Of particular 
interest to the TEEODE project were the methods of assessment and evaluation employed by the various suppliers of 
distance learning courses. The aim of the TEEODE project was to develop a representative survey of the models of 
assessment and evaluation used by institutions supporting distance learning. This survey endeavoured to cover all 
fifteen member states of the European Union. 

For each of the member countries of the EU we attempted to survey up to 10 institutions on their formal procedures 
or framework for assessment and evaluation; and up to 10 tutors on individual courses within those institutions on 
how they conduct their assessment and evaluation. 

In this part of the TEEODE project we have focused upon issues of Course Development and Assessment, and in 
particular, on how we achieve effective learning outcomes and how that learning is communicated to the learner. 
Other parts of the project have looked more closely at the institutional issues. The emphasis here is on the tutors’ 
perceptions, teaching assessment and support-related issues. 

If distance learning is to be seen as legitimate it must meet student, tutor and, in many cases employers’, needs. In 
assessing their students, tutors should have a clear idea of their learning goals and they should also be able to match 
those goals to the assessment and evaluation instruments they employ. 

Marton and Saljp (1976) showed that students that tackled their studies in a way which was called a ‘deep approach’ 
gained a through understanding of the material studied. On the other hand, students whose approach was labelled a 
‘surface approach', failed to gain a grasp of the material which had been studied. Ramsden (1988) provides a useful 
summary of ‘deep’ and ‘surface’ approaches to learning. 



Deep Approach 
Intention to Understand 



Surface Approach 
Intention to complete learning task assignments 



Focus on what is ‘signified’, 


Focus on ‘signs”. 


i.e. the author’s arguments 


i.e. the text itself 


Relate and distinguish new ideas 


Focus on discrete elements 




Memorise information 


Relate concepts to everyday experience 


Unreflectiveley associate concepts and facts 




Fail to distinguish principles from evidence, new 
information from old 


Organise and structure content 




Learning is internal 


Treat the learning task as external imposition focusing 
on the assessment 



In the tutor questionnaire presented here we were interested in identifying tutors" perceptions of: 

1. the learning goals of the individual course they chose to tell us about; 

2. the modes of assessment they felt aided them in achieving those goals; 

3. any mismatch that might occur between those goals and the modes of assessment. 



1 The Course Tutor Questionnaire 

The Course Tutor Questionnaire (CTQ) was based on interviews with tutors and administrators in ODE institutions, 
and the final versions (pilot as well as translations) were also checked for readibility and understandability. 

Translations of the English version were made into Finnish, French, German, Italien, Portuguese, Spanish and 
Swedish. 

In the process of translating the English version, a number of problems were encountered which had to do with 
differences in the educational systems in the different European countries. What, for instance, is the equivalent of an 
English BA in other European countries? We endeavoured to approximate to the English original as close as possible 
to make results comparable across European countries. 

The final form of the CTQ consists of seven parts which relate to 

1 . the tutor, 

2. the course, 

3. the students, 

4. the assessment of the course, 

5. production of the assessment material, 

6. delivery of the assessment instruments, 

7. staff development. 

If the respondents were interested in receiving a copy of the results of the investigation, they were asked to provide 
their address. 

The different versions of the CTQ were made available in the internet at 
http://www.doe.d5.ub.es/te/teeode 

The user was asked to type in his or her responses and these were then collected at the the University of Barcelona. 
Also, paper versions of the CTQ were sent to ODE institutions in different European countries. 



2 The Sample 
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The initial sample of 10 tutors from each of 10 institutions for each of the member countries of the EU we attempted 
changed over time. One of the main reasons for this was the method of data collection (paper-based questionnaires 
sent out by regular mail and on-line questionnaires made available on the web). 

There were two main benefits of the on-line questionnaire. The first was that it widened the sample beyond the initial 
fifteen member states, hence the inclusion of data from Israel, Mexico and the USA in our survey. It also proved 
highly efficient in gathering data together and the transfer from questionnaire to data-sets for analysis was achieved 
with efficiency through this route of data collection. 

Unfortunately, however, there were some problems with accessing the on-line questionnaires and we have suffered 
data loss, thus some 25% of the UK sample is missing and we are aware that our Scandinavian contacts also had 
difficulty in initial access to the web questionnaire. 

We tried to reach as many ODE institutions as possible, being well aware that, within the limitations of our budget, it 
was not possible to have neither a truely representative nor a purely random sample. We did hope, however, that we 
might achieve representativeness at least to some degree. 

There were 224 CTQs returned. They were distributed across the different countries very unevenly. It is difficult to 
establish the reason for this, but one explanation that seems very plausible is that ODE institutions in Europe vary 
very much with respect to the extend and importance they have in the different European educational systems. In the 
UK ODE is well established but this is not the case for many of our European partners. 



3 The tutors 

Of the 224 respondents, 59 (26.3 % of 224) indicated they were course organizers and 175 of them (78.1 % of 224) 
declared to be course tutors. However, only 17 (7.6 % of 224) acted both as course organizers and course tutors. 

Somewhat more than half of the tutors were employed on a full-time basis (125 versus 99 half-time, i.e. 55.8 % 
versus 44.2 %). An interesting finding was that all of the tutors were relatively new in their courses: none of them 
had worked for more than three years in his course, 99 (44.2 %) had given his course for less than three year but for 
more than one year, and 125 (55.8 %) had given his course for less than one year. This finding may reflect that rapid 
and recent rise in interest in ODE across our institutions or, it may be the case that tutors selected to talk about new 
courses rather than courses they have worked on for a number of years. 

4 The courses 

Table 1 gives an overview of the subject areas that were covered by the different courses. The single most cited 
category is languages (33.5 %), followed by literature (20.5 %). If we subsume the first six categories under the 
heading "Science of Letters", than almost three quarters (71.4 %) of the courses fall under this heading and a 
remaining 18.3 % (categories 8 to 12) under the heading natural sciences. This contrasts sharply with the findings of 
the Institution Questionnaire in which Business and Management Studies dominate the ODE field. 





Frequency 


Percent 




Literature 


1 


46 


20.5 


Social Sciences 


2 


9 


4. 


Languages 


3 


75 


33.5 


Education 


4 


19 


8.5 


Humanities 


5 


11 


4.9 


Arts 


6 


- 
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Medicine 


7 


1 


.4 


Math & Statistics 


8 


12 


5.4 


Business and Management Studies 


9 


11 


4.9 


Engineering 


10 


- 


- 


Computer Science 


11 


5 


2.2 


Natural and Physical Sciences 


12 


13 


5.8 


Others 


13 


22 


9.8 



Table 1: Subject areas of ODE courses 



Of the 224 tutors, 175 (78.1 %) rated their courses as academic and 45 (20.1 %) as professional or vocational. The 
length of the different courses varied substantially. 68 (30.4 %) of the tutors indicated that their courses lasted less 
than a year, 81 (36.2 %) had courses lasting between one and two years, 20 (8.9 %) were involved in courses lasting 
more than one year but with a time limit, while 55 (24.6 %) indicated that their courses did not have any time limit. 

Table 2 shows how the tutors rated the level of their courses. As is evident, the vast majority of the courses (170 of 
224, i.e. 75.8 %) lead to an academic qualification. This is very much in line with the tutors' labeling the courses as 
academic versus professional or vocational. 





Frequency 


Percent 


Postgraduate degree 


34 


15.2 


Bachelor degree 


21 


9.4 


Certificate or diploma 


115 


51.3 


Professional qualification 


14 


6.3 


No qualification 


8 


3.6 


Other 


30 


13.4 


No answer 


2 


.9 



Table 2: Level or courses 

Cross-tabulating subject areas by course level shows that 63 of the 74 language courses (85.1 %) and 37 of the 47 
literature courses (78.7 %) are academic while of the remaining subject areas in the sample , 23 of 58 are (71.6 %) 
academic. 



5 The students 

We asked for the age of the ODE students, but the results were rather inconclusive. 130 tutors (58 %) indicated that 
there was no set age and 69 (30.8 %) judged the age of their students to be more than 23 while only 25 (11.2 %) 
assumed their students to be younger than 23. It therefore seems that the age of the ODE students of our sample may 
be somewhat higher that that of "normal" college and university students. This is consitent with a view of ODE as an 
important way of gaining qualifications while working. 
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Concerning the entry qualifications of the students for the ODE courses, about half of the courses require no prior 
qualifications while requirements for qualifications are almost evenly distributed across the categories GCSE or 
General School Certificate (post 16; 15.6 %), A-level or Advanced School Certificate (16.5 %) and Bachelor Degree 
or equivalent (17.4 %). 



6 Assessment of the courses 

Of the 224 tutors, 92 (41.1 %) reported that formative assessments were undertaken during the course, 16 (7.1 %) 
indicated formative as well as summative assessments were being conducted while 116 (51.8 %) reported that 
formative as well as summative assessment was being done. 

The most preferred mode of assessment turned out to be written assessment (208, corresponding to 92.9 %) 
followed by practical assessment (51 / 22.8%) and oral assessment (43 / 19.2 %). 

Please note: For this and the following questions in this part of our report, the alternatives were not mutually 
exclusive, i.e. a tutor might indicate that the assessment was conducted in written form as well as orally and 
practically; percentages are based on the 224 responses for each alternative. 

With respect to the question of where the assessment was conducted, 84 tutors (37.7 %) named the host institution, 
86 (38.4 %) a designated centre at a distance, and 49 (21.9 %) a place of the student's choosing. 

The tutors were given a number of alternatives which described the main objectives for assessing students on their 
courses and asked which of these they deemed most appropriate with respect to their own courses. Answers are given 
in table 3. 



to test factual knowledge 


71 


(31.7 %) 


to test understanding 


145 


(64.7 %) 


to test application of knowledge 


167 


(74.6 %) 


to test practical skills 


44 


(19.6 %) 


to test problem solving abilities 


52 


(23.2 %) 



Table 3: Main objectives for assessing students 



Evidently, the tutors consider application of knowledge and understanding as the most important objectives for 
assessing students' achievements. 

In order to learn more about tutors' motives and beliefs with respect to conducting assessments of their students, we 
asked them to what extent different forms of assessment matched their educational goals. The forms of assessment 
were (1) multiple choice questions, (2) comprehension tasks, (3) problem solving tests, (4) practical tests, (5) project 
file or diary, (6) oral presentation or examination, (7) long essay or dissertation, and (8) written examination. 

The tutors were asked to indicate to what extent the forms of assessment matched their educational goals on a five- 
point scale: (1) excellent, (2) good match, (3) satisfactory, (4) poor match, (5) no opinion. From these data, we 
computed two indices. First, we defined the degree of acceptance (DA) as the proportion of tutors (in percent) who 
chose to check one of the first three alternatives, i.e. who indicated that they perceived at least a satisfactory degree 
of match between the form of assessment and their educational goals. 

Second, we transformed these categories (degree of match) into numerical values, assigning 1 to excellent, 2 to good 
match etc. We then computed a mean rating (MR) for those tutors who did rate the match between form of 
assessment and their educational goals. A low MR thus signifies a high degree of acceptance, and conversely, a high 
MR signifies a low degree of acceptance. 
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The most acceptable form of assessment (79 %) turns out to be the assessment by comprehension tasks, marked by 
the tutor and conducted for formative assessment. This form is also rated highest with respect to the match between 
assessment form and educational objectives. These results are in line with our former findings (see table 3) that the 
tutors considered the assessment of understanding and the application the main objectives of their assessment 
procedures. Interestingly, this form is less acceptable for summative assessment purposes (55 %) or with students or 
peers doing the marking (38 % for formative and 22 % for summative purposes). 

It seems that the question of who does the marking has a decisive influence on the tutors 'ratings. For example, with 
respect to multiple choice questions, the best match between form of assessment and educational objectives is 
achieved when the marking is done by the tutor; students' marking obviously widen the gap between assessment and 
educational objectives, while computers seem to be even less welcome as marking agents (the mean rating for this 
combination is 3.11 - formative - and 3.16 - summative assessment). Preference for tutors' marking is also exhibited 
with respect to the other assessment forms (indicated by the degree of acceptance as well as by the mean ratings). 
Althoug our tutors are working in a new mode of teaching, they are still using very traditional approaches to 
assessment. 

Finally, with the exception of the „classical“ assessment forms of written examination, the assessment instruments 
are considered more acceptable for formative than for summative assessment. 



7 Production of the assessment material 

In this section, we wanted to know where the assessment material that the tutors are using come from. More than half 
of the tutors (126, i.e. 56.3 %) indicated that were using assessment instruments that were produced by course team 
members. In most cases (98, i.e. 43.8 %) the instruments used were the result of a collaboration of the course team. 

However, some tutors to some extent (57, i.e. 25.4 %) used assessment instruments that were not produced by course 
team members most of which (67 %) came from commerical sources. 

Concerning the type of the assessment instruments, 64 (28.6 %) of the tutors used instruments that were standardised, 
80 (35.7 %) used instruments that were not, while 75 (33.5 %) did not know whether their instruments were 
standardised or not. 



8 Delivery of the assessment instruments 

While in the section on the assessment of courses we had asked for the educational objectives that were being 
persued in assessing the achievment of the ODE students and to what extent the instruments being used matched 
these educational objectives, in this section we wanted to know what kind of assessment instruments were actually 
being used. 

When asked about the “traditional" instruments, 61 tutors (27.2 %) reported using oral tasks and 206 (92.0 %), i.e. 
the vast majority used written tasks. However, only in a very few cases was this assessment computer supported (in 3 
cases for oral tasks and in 19 cases for written tasks) . 

Tutors were also asked if there existed a set interview schedule that must be followed in oral examinations. 33 (14.7 
%) responded yes, and 30 (13.4 %) responded no, while 155 (69.2 %) checked the do-not-know alternative. This is 
another indication that oral examinations do not seem to be used frequently as assessment tools. 

In general, assessment criteria do exist, however. 90 (40.2 %) tutors declared this to be true, only 10 (4.5 %) denied 
this, and 1 17 (52.2 %) preferred to check the no-answer alternative. 

Finally, we wanted to know if computers were used in the assessment and in what ways. Very few tutors used 
computer technology for assessment. Of the tutors that did so, 7 indicated that the system marked students' reponses 
while 21 indicated that it did not. At the same time, 13 reported that the system provided feedback on reponses to 
students while 12 reported that it did not. 
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Of those tutors who used telematics, 6 conducted assessment in real-time and on-line while 16 did not conduct the 
assessment this way. 



9 Staff development 

In this last section, we wanted to find out in what areas there might be a need for further advice or training, and we 
asked the tutors whether they felt they might benefit from advice or training in the areas given. Frequency counts of 
the answers are exhibited in table 4. 





Introductory training 


Advanced training 


Design of distance learning materials 


25 (11.2%) 


51 (22.8 %) 


Assessment of distance learning 


11 (4.9%) 


39 (17.4 %) 


Design of multiple choice tests 


16(7.1 %) 


31 (13.8 %) 


Design of problem solving tests 


18 ( 8.0 %) 


32 (14.3 %) 


Marking of essays 


7(3.1 %) 


21 (9.4 %) 


Design of formal examinations 


7 (3.1 %) 


24 (10.7 %) 


Use of computer based tests 


37 (16.5 %) 


39 (17.4 %) 



Table 4: Areas in which advice and/or training would be appreciated by the tutors 



The data show clear variations in tutors" perceptions of their personal training needs for different modes of 
assessment. Few tutors felt that they needed additional help in marking, essays or in preparing formal examinations. 
However, significant numbers requested help in computer based assessment at both an introductory (16.5 %) and 
advanced (17.4 %) level. There was also considerable interest in obtaining additional training in the design (22.8 %) 
and assessment (17.4 %) of distance learning courses - but this need was for advanced level skills. 



10 Summary and conclusions 



The returns of the Course Tutor Questionnaire did not live up to our expectations. We invested a large amount of 
time and effort to contact ODE institutions in Europe and to distribute the questionnaire, but we received a relatively 
small number of responses. We only know that return quotas are in general relatively low with this kind of studies. 
Additional data has been received post this analysis. They will be incorporated in the year 2 report. 

Nonetheless, from the tutors who did respond we received some very interesting data. It is clear that tutors aim at 
testing the application of knowledge and understanding when they conduct their assessments, and there is a 
preference for formative assessment, i.e. assessment that is done during the course. Traditional methods are being 
used, but tutors" preference is with comprehension tasks, multiple choice questions, and course essays. At the same 
time, tutors feel they would benefit from advice and training in the design of distance learning material and in the 
assessment of distance learning. 

With respect to the computer, the results seem to be a bit paradoxical: On the one hand, tutors indicate that their 
educational goals can better be reached with multiple choice tests when marked by the tutor than when marked by 
the computer. This is difficult to understand because it would not at all be difficult to do the marking of multiple 
choice tests with the a computer program in an “objective" manner. The objectiveness of multiple choice tests is in 
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general very high because the assessment of the results is independent of the person (or machine) who does the 
marking (given that there is a key for the correct alternatives). We therefore conclude that many tutors still do not 
feel at ease with the idea of letting the computer do the assessment simply because of lack of familiarity with this 
possibility. We consider this assumption to be substantiated by the fact that more than half of the tutors also were 
relative new in their teaching jobs (with an experience of a year or less with their courses) and that at the moment, 
there is indeed relatively little use of computers for assessment purposes in ODE institutions. 

At the same time, tutors would appreciate advice and training in the use of computer based tests. Since they also 
exhibit interest in getting to know more about the design of distance learning materials and in the assessment of 
distance learning, we assume that their somewhat hesitant attitude towards computers would turn into a favourable 
one if they actually were given advice and training in using computer programs to develop distance learning material 
and instruments to evaluate the learning processes of their students. 
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